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ABSTRACT 



Professors and courses were evaluated sociometr ically by almost 90% 
of all full-time resident students. The three professors and courses 
they would most as well as least recommend to other students were listed. 

A 30% random sample of respondants was then interviewed by other 
students. Both the sociometric results and the results of the interviews . 
were given to faculty residents during the summer (approximately 70%). 

They were asked to respond to a questionnaire primarily devised to indicate 
credibility and defensiveness. Half the faculty were given coded 
interview results and half verbatim results. No significant differences 
in response were found. Also, there was some evidence that the 
procedure produced changes in faculty attitudes toward themselves and 
their courses. Results were also scrutinized as to tenure or none, 
years at Austin College, rank, age, choice quartiles (high most-high 
least, high most-low least, low most-high least, and low most-low 
least), highest degree earned, and area (Humanities, Social Sciences, 
or Physical Science). Results were sensible and the null hypotheses 
disconfirmed. Results and implications were discussed. 
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DEVELOPMENT AND VALIDATION OF A SOCIOMETRIC INSTRUCTOR 



EVALUATION INSTRUMENT AND PROCEDURE 

The evaluation of instruction, either by self or other, is fraught 
with fear, distortion, conflicting objectives, rubber yardsticks, etc. 

Very little scientific progress has been forthcoming in this area so that 
instruction is best termed an art rather than a profession due to the 
lack of procedures for professional self correction. Within very broad 
limits, teachers teach as they see fit so that they live in an academic 
hothouse shielded from the chilling winds of negative upward communication 
as well as precise, reliable negative downward communication. This 
environment simultaneously nurtures ego delicacy from lack of negative 
feedback as well as attracting people who find this environment friendly 
to their own personal needs. 

Instructional effectiveness has constituted an area of concern since 
the Sputnik era when Ivan was thought to know more than Johnny, sparking 
the development and institution of massive currciulum enrichment and 
modernization accompanied by heavy handed pressures put on our young people 
to learn more, achieve more, and generally perform better in numerous ways. 
These changes in curriculum and pressures on students were to no avail 
without provisions for excellence in instruction and its inevitable 
counterpart-teacher evaluation. During the more recent years of student 
protest that may have been related to these post-sputnik pressures, two 
words resounded from many directions - governance and relevance. The 
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latter term further implies the need for better evaluation of instruction. 

The most recent development seems to be the "new education" movement which 
has accompanied or represents a reaction to an oversupply of teachers at 
all levels. Whatever the cause and effect relationship it is simple 
logic to predict a search for effective evaluation procedures. Shinn 
(1972) seems to describe the "new education" as more action oriented, 
collaborative or student composed, humanistic' and simultaneously technological, 
competance oriented, continuing, process oriented, questing or curiosity 
oriented, etc. These represent or encompass objectives which are not new 
but newly or more emphatically sought after. But to many such objectives 
represent a counter culture in relation to the way the teachers, of *' 
today were taught* and hence misunderstood or resisted by faculty and 
administration alike. 

Many attempts were made to nail down teacher or institutional 
objectives (Gage, 1960). Tyler (1960) states that: 



"In the American college community the evaluation 
of teaching is as common as the judgment of the 
quality of dormitory food and often as subjective. 
'Professor Smith is a wonderful teacher!' "We are 
a select college, proud of our tradition of excell- 
ence in teaching.' Comments like these are part of 
the typical pattern of college conversations. Yet 
we know they are not highly valid, objective, and 
impartial appraisals. Sound and systematic evalua- 
t,i° n of college teaching exceedingly hard and yet 
it is highly essential to the improvement of 
college instruction." 



Inherent in the idea of excellence, quality, or improvement is the 
concept of objectives. These are the benchmarks next to which in- 
struction has to be evaluated. Yet when the objectives of an effective 
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teacher or even a definition is sought we find taxonomies (Bloom, 1956) 
or subjectively conceived student questionaires. General impressions 
on reading Harvard's annual Confidential Guide or the University of Texas' 
Devil ' s Report Card are that they are "cute" journalism. But these, and 
other efforts serve to accentuate the timeliness and need for effective 
instruments for evaluation and feedback. 

industry constitutes social systems with outputs which are quite 
tangible and easily measured. Consequently studies on the variables 
related to productivity are numerous and perhaps suggestive of possible 
strategies to employ in educational social systems. Performance appraisal 
ss widely used in industry (basically a hierarchical approach characterized 
by downward communication) has many difficulties associated to which teacher 
evaluation can easily fall prey. Blake (1952) comments that: 



"When it comes to evaluating people, though, even 
the most skillful manager loves his perspective and 
turns into a demon. He gets, out his rating forms, 
with viscious delight combined with pain and anxiety, 
goes to work. He answers, to his own satisfaction, 
the 32 items on the form about his subordinate including 
such evaluations as dependability, initiative, neatness, 
ability to learn, promptness, character, adaptability, 
resourcefulness, enthusiasium, loyalty, judgment, in- 
tegrity, decisiveness, tact, and 19 others. . .He 
repeals his good sense and starts acting like a 'God', 
a clinical psychologist or a psychiatrist. . ." 



Likert (1961) has also addressed himself to evaluation of the first line 
supervisor which could be roughly equivalent to the college teacher. His 
procedure takes the threat out of the situation by taking evaluations 
from subordinate and th-? information available only to the supervisor 

being evaluated. First-line supervisor data is only given to superiors 
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averaged in with a group of fellow supervisors. He can then decide if 
he is perceived as the kind of supervisor he would like to be — all without 
the threat and defensiveness inherent in other procedures. Real change 
is perhaps more likely to occur under such a system. 

In faculty evaluation instruments, characteristics could be dwelt 
upon which mightnot apply to an instructor's strategy for instruction, 
a.g. , lecture oriented items applied to a laboratory-inquiry oriented 
course, or vice versa. An excellent set of catagories was devised by 
Hildebrand and Wilson (1970) using impeccable scientific procedures, the 
best correlation and factor analytic techniques. The resulting thirty- 
six items placed into five groupings (Appendix A) ostensibly measure 
effectiveness of instruction, but they would probably seem austere and 
unimaginative to the creative college instructor, i.e. representing 
convergent thinking in its most mathematical sense when he is faced with 
an increasingly diverging world in which walking dictionary-type college 
graduates quickly find themselves obsolete. 

One is challenged, therefore, to approach the problem in a new or 
different way, free from the entrapments contained in conventional methods 
and procedures for measurement and evaluation of effectiveness of 
instruction. A major objective suggested by the state of this literature 
would be the development of an instrument to measure effective instruction 
while being sensitive to unique, individualized objectives of specific 
instructors. Such an instrument, coupled with an effective procedure 
for individual or group self-correction might constitute a needed advance 
in social science technology as related to educational institutions. 
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Such an approach arising from Moreno's (1953) Hudson Girls' School 
socioraetric study coupled with a non-threatening evaluation procedure, 
seems promising* Threat is removed by making the information available 
to the professor only, leaving the administration to fend for themselves 
to meet their evaluative needs with whatever criteria they deem appropriate* 
The use of sociometric data provides information as to how the teacher 
rates according to the student-centered criteria rather than criteria based 
on the values of the administration, a particular discipline, any current 
relevance fad, or any lovely cognitive design of some expert or consultant. 
The students av<v treated more as clients than clay and trust is placed 
in the individual teacher to apply self-corrective measures in response 
to negative upward communication hitherto unavailable or garbled. A 
sample of students could then be interviewed as to why they made the pro- 
fessor and courses choices they made, and this data refined into codes 
so as to render the interviews unnecessary for the next time the instrument 
is used. 

Hypotheses . Null hypothesis 1: Faculty survey responses of those 

receiving coded interview responses will not differ from those receiving 
verbatim interview responses. Null hypothesis 2: Attitude survey results 

and other response of faculty will not give indication that the data and 
procedure are instrumental in producing change in courses or methods of 
instruction. Null hypothesis 3s Attitude survey results of the faculty 
divided by age at the fiftieth percentile (39 or less— 40 or more) will 
not differ significantly. Null hypothesis 4; Attitude survey results 
of faculty having tenure will not differ significantly from those not 
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having tenure. Null hyporhwaia 5: Attitude survey responses of the 

faculty divided by number of years at Austin College (five years or less-- 
six or more years) will not differ significantly. Null hypothesis 6: 
faculty attitude survey responses of the faculty will not differ sig- 
nificantly according to academic rank (Instructor or Assistant Professor-- 
Associate of Full Professor). Null hypothesis 7; Faculty attitude 
survey responses will not differ significantly by academic subject 
matter divisions (Humanities, Social Science, or Physical Science). 

Null Hypothesis 8: Attitude survey response of faculty will not differ 

significantly according to Most-Recommended--Least-Recommended quartiles 
(High-Most and High-Least* High-Most and Low-Least, Low-Most and High- 
Least, or Low-Most and Low-Least.) Null hypothesis 9: Attitude survey 

results of the faculty will not differ significantly according to highest 
degree earned (B.A. or M.A. vs Doctorate). 

These hypotheses are indicative of the major objectives of this 
research, namely: (1) to develop an effective sociometric faculty 
evaluation instrument which will not only indicate positive and neg- 
ative choices but effectively and inexpensively gives some indication 
of the resons students make these choices, and (2) to develop a low 
threat procedure for communicating the data in such a way that is likely 
to produce change in courses and methods of instruction. 
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EXPERIMENTAL DESIGN 



Previous Research Groundwork 

Over the five years pervious to this experiment, a sociometric 
instrument has been under developments Only last year were there efforts 
to communicate the results of the works When first administered, the 
questions were part of a more comprehensive questionnaire given to a 
ten percent sample of the student body. While random sampling seemed 
adequate for attitude survey questions, doubts arose as to the reliability 
of such results in relation to a sociometric studyo The following year 
a more comprehensive coverage was attempted * 

During the last three years, the sociogram became an annual Spring 
research project for students in the introductory sociology course and 
between 80 and 95 percent of all full time resident students responses 
to a sociometric questionnaire were collected. (Appendix B) Four 
questions were asked of as many full-time resident students as could be 
contacted: (1.) "Would you please list the three professors you would 

most likely recommend to other students, (2.). . .the three professors 
that you would least likely recommend to other students, (3.). . .the 
three courses that you would most likely recommend to other students, 

(4.). . .the three courses that you would least likely recommend to 
other students." These questions constituted the dependent variables. 
Independent variables include! sex, classification (freshman, sophomore, 
junior, senior), major, Greek or non-Greek and which organization if 
Greek, and finally, residence. In previous years four print outs were 
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executed by the computer: (1.) a raw score rank order of professors 

most recommended, (2.) a raw score rank order of professors least 
recommended, (3.) one of courses most recommended, and (4.) one of 
courses least recommended* One question which continually arose was 
the matter of the reasons why students made the various choices they 
made. 

This question frequently arose in the Spring of 1971 when the 
results of the sociometric instrument were released to faculty members 
who volunteered to receive the results and comment on them. Previous 
to this time, it was fairly common knowledge around campus that this 
research was going on, but results were not being released. Frankly, 
the volatility of the data led not only to stringent precautions 
against compromising of the data but also some reluctance to distribute 
the results. 

Sociometric data was collected as previously described by intro- 
ductory sociology students in April of the 1972 Spring term. Each 
finished response sheet was coded by numbers assigned to each student 
on any particular dormitory floor so that the names of the students 
could be located for purposes of interviewing them. The response sheets 
were then assigned consecutive numbers from which a. 30 percent sample of 
interview subjects was drawn from a table of random numbers. (Rand 
Corporation). Table 1 indicates the degree of randomicity the sample 
achieved . 
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TABLE 1 

REPRESENTATIVENESS OF INTERVIEW SAMPLE 
TO ENTIRE POPULATION 



Population Characteristic 
Total Responding 


Entire Population 
755 


Sample 

254 


Percent of Population 
33.64% 


Sex 








Females 


345 


118 


34.20 


Males 


410 


136 


33.17 



Classification 



Fre shmen 


307 


109 


35.50 


Sophomores 


203 


56 


27.59 


Juniors 


144 


51 


35.42 


Seniors 


96 


36 


37.50 



Affiliation 








Greek 


281 


101 


35.94 


Non -Greek 


474 


153 


32.27 



Major 








Humanities 


175 


67 


38.28 


Physical Science 


176 


56 


31.81 


Social Science 


231 


70 


30.30 


Undecided 


91 


30 


32.96 


Mixed 


151 


31 


20.52 



! 

J 

j Interviewers . Twenty upper division undergraduates who had experience 

with action projects associated with other sociology courses were invited 
to attend a three hour training session and to conduct open-ended inter- 
views on these sociometric respondents included in the 30 percent random 
sample of those respondents. The interview training session was centered 
around a short lecture and role-playing of interviews. Three basic 
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points were made in the lectures (1) it is basically a compliment to 
ask someone their opinion, (2) a good interviewer keeps listening when 
others have stopped, and (3) when those being interviewed look to the 
interviewer for cues as to how they should respond they should be reminded 
that it is their views and opinions that are wanted* Mention was made 
of how most people will search for a frame of reference from others when 
they are uncertain as to what constitutes appropriate behavior with 
examples given of how Rohrshach responses had been experimentally induced 
simply by voice intonations used in "hram" or Uh-huh" responses by the 
tester. At this point, questions concerning how to draw people out led 
almost imperceptably to role playing situations on how to draw out more 
tangible and explicit responses when asked why they chose either professors 
or courses as those they would most or least likely recommend to other 
students. This shift was subtle but provided a good bridge to the practice 
interviewing. First, they interviewed themselves and then paired off 
to practice interviewing another person in the training session. Finally, 
questions were fielded concerning the practice interviews which mainly 
were answered by the basic points made in the lecture-compliment when 
asking opinions, nonstop listening, and guarding against suggesting what 
constituted an appropriate response« 

Coding. Interview responses were then subjected to the standard 
information processing technique of catagory development by reading the 
interview comments one by one, developing each category as needed, and 
combining categories when appropriate. This procedure was used for all 
four of the sociometric questions that were asked. The results were four 
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lists of codes-one for each question: professor most recommended, least 

recommended, and courses most and least recommended (see Appendix C). 

Subjects . The primary focus of the study was on the potential of 
the instrument and procedure for producing change in faculty members. 

The sociometric lists and interview results were given to all faculty 
members on campus this summer (1972) and they were given an attitude 
survey questionnaire to assess their reactions to the experiment. An 
example of all materials included in the faculty results packet (see 
Appendix C). Half of the faculty were given coded results and half 
given verbatim interview comments on themselves and the courses they 
taught. An attempt was made to allow for a full range of response 
alternatives in the questionnaire with twenty-four Likert type questions, 
one nine point question on the value of student opinion, and finally 
an open ended question which simply stated: "your comments are earnestly 

solicited." Faculty responses were subjected to Chi-square statistical 
tests concerning eight independent variables: (1) those receiving coded 

vs verbatim interview responses; (2) tenured or not tenured, (3) years 
at Austin College; (4) rank--instructor, assistant professor, associate 
professor, or full professor; (5) age — 0 to 50th percentile or above; 

(6) least-most recommended choice quartiles--high-most - high. least, 
high most-low least, low most-high least, and low most-low least; 

(7) highest degree earned--B.A. and M.A. level or Doctoral level) and 
finally, (8) area of division--social science, physical science, or 
hummanities. Comments were treated as well as the mean of the responses 
even though a measure of control tendency assumes interval rather than 
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ordinal data--an assumption which some criticize and others accept. 
This matter is treated more precisely in the next section. 



RESULTS 



Coded or Non-Coded Responses 

Since the primary thrust of this study, as suggested by the title 
of the study, was the development of a sociometric instrument for 
faculty evaluation, the Chi-square Contingency tests on the faculty 
attitude survey questionnaire (see Appendix D) between those faculty 
who received word for word or verbatim interview results (an example 
is given as a part of the packet all faculty were given (see Appendix 
C), and those faculty who received coded response (example also 
included in the packet given the faculty (see Appendix C). Approximately 
half the faculty were given coded responses (N=j34) and no significant 
statistical differences were found on the faculty attitude survey 
questionnaire (see Appendix D). Consequently the first null hypothesis 
that there is no difference between faculty receiving coded vs verbatim 
interview responses as indicated on the faculty attitude survey results, 
is accepted. This gives strong confirmation that interviews will be 
rendered unnecessary by a sociometric questionnaire incorporating the 
coded alternatives. 

Producing Change 

The second null hypothesis states that the faculty attitude survey 
results and other faculty responses will not give indication that the 
data and procedure were instrumental in producing change in course or 
methods of instruction. A firm rejection of the second null hypothesis 
is quite difficult to achieve since most of the evidence is quite 
qualitative in nature. Although the experimentor has no desire to enter 
into the continuing dispute as to whether SD-SA Likert type scales may 
be treated as interval scales or only ordinal scales, Table 2 utilizes 
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a measure of central tendency (the mean) so that results can be generally 
interpreted as being reacted to by the faculty with disagreement, indecision, 
or agreement. The mean was calculated by assigning values to each response: 
Strongly Disagree = 1 , Disagree = 2 , No Opinion or No Response = 3 , Agree 
= 4, and Strongly Agree = 5. The mean was interpreted as "Disagree 11 
if it fell between 1.0 and 2.49, "Indecision "be tween 2.5 and 3.49, and 
as "Agree " between 3.5 and 5.0. 

In general, using this central tendency as an indicator (Table 2), 
it may be said that faculty disagreed with question numbers 1, 5, 9, 19, 
and 27. Three of these statements were designed to give respondents a 
chance to indicate threat regarding the data: (1) I don't care who it 

is, it makes me \ervous knowing anyone has information like this - 
mean of 2.19 (disagree), (5) This data-taking should be stopped - mean 

| of 2.17 (disagree) and (19) I find this whole experience degrading - 

r 

i 

i 

i mean of 2.14 (disagree). 

Two of these statements were designed to give the respondents a chance 
to indicate their feelings concerning the value of the data. (22) The 

i 

j choice scores and student comments do not provide enough basis for any 

j- kind of corrective action on my part - mean of 2.26 (disagree) and 

j (24) Student comments were useless to me - mean of 1.85 (disagree). 

I 

Three statements were generally agreed to regarding the value of the 

5 . 

i 

i data. (4) This information helps me evaluate myself - mean of 3.58 

| (agree), (8) I would like to get confidential information like this 

5 

| every year - mean of 3.55 (agree), and (13) Student opinions are important 

to me - mean 4.02 (agree). 
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Of those statements dealing with threat, six showed results of in- 
decision: (2) I find this information very informative - mean of 3.35, 

(3) I didn't learn anything about myself or my courses from this that 
I didn't already know - mean of 2.82, (11) I think I can improve my 
teaching due to my receiving this information - mean of 3.17, (16) Student 
comments make the student recommendations of me and my courses more 
understandable - mean of 2.94, and (23) The student comments constitute 
the best information in the whole "package" - mean of 3.44. Several 
questions regarding threat were responded to indecisively: (10) I wouldn' 

mind if this information were made public - mean of 2.71 (bimodal 
distribution), (12) I'm not sure the administration would be able to view 
this information in the proper perspective - mean of 3.04 (bimodal 
distribution), and (14) I wouldn't mind if the administration used this 
information as a part of their evaluation of my performance or my 
effectiveness as a teacher - mean of 3.26. 

The remainder of the questions dealt with confirmation of such matters 
as whether respondents could read the computer printouts (question 9) 
or whether more precise data should be attempted with the application 
of a student exposure correction faction (questions 20 and 21). Con- 
sequently, the second null hypothesis was either disconfirmed by faculty 
attitude survey responses or yeilded indecisive responses, but in no 
case did the indication of central tendency confirm the null hypothesis 
that the data would not indicate that sociometric data and procedure were 
instrumental in producing change or methods of instruction. The final 
nine point question showed no significant differences no matter how the 
respondants were divided. However, all seemed to indicate some value to 
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student opinion although the distribution (see Appendix M) seems to 
indicate the question was poorly designed. 

Qualitative Evidence . More difficult to evaluate is the qualitative 
evidence which should be applied to the consideration of this hypothesis. 

The most dramatic event occurred when a very "though-minded" member of 
the physical science division got up before the entire faculty on campus 
this summer (approximately 70 %) and stated that he must change or become 
obsolete and that he was motivated primarily by the sociometric and 
interview data treated in this report. He stated that he found out he 
was not a good lecturer as he had thought and consequently was adopting 
a modular approach emphasizing self-paced learning. 

Also, application for tenure or promotion at Austin College includes 
the preparation of lengthy answers to several questions for review by 
a committee of the Board of Trustees. Two such applications included 
the sociometric and interview data as indicators of their teaching 
effectiveness. 

In addition, one of the deans commented that the data had evidently 
had a salutary effect on many professors who had not otherwise indicated 
that such was the case. Another area dean uhedptheoproaeduderQQdlcesulbts as 
a case study in his role as a consultant to a seminar for new heads of 
physical science departments, conducted by the Research Corporation to 
be noted in the "Proceedings" of the conference (in publication). 

Finally, several faculty members personally expressed their 
appreciation for the insights they received from the data they had 
received. This was gratifying to the experimentor who during the period 
of dissemination, regularly sniffed the air for the odor of melting tar, 
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and listened for the sound of chickens being plucked of their feathers. 
This is not to say that one or two faculty members did not react in a 
negative way to their "package." This can be seen by the comments (see 
Appendix D) and by the refusal of six subjects to react to the data by 
means of the faculty attitude survey questionnaire. Follow-up requests 
were made several times asking for the return of the questionnaire 
simply marked "refused" if they did not wish to fill it out. As a last 
resort, telephone calls were made with as low pressure a tone as possible 
indicating that it was simply a bookkeeping sort of request. If anyone 
showed the least bit of resistance they were thanked and told that any 
reservation they might have was respected but if they had any comments, 
they would be appreciated. 

Considering the results of the means and the more qualitative results 
and events, the second null hypothesis appears to be largely confirmed 
except for a small minority of the faculty who indicated they did not 
place much value on the results or were threatened by those results. 

Age . Faculty members were divided by age at the fiftieth percentile 
and Chi-square contingency tests were applied to the responses to the 
faculty attitudes survey questionnaire. No significant differences were 
found except for question 17, I suspect students recommend easy courses 
and professors. On this matter, the younger half of the faculty (39 or 
under) tended more to agree with this question (P = .05) (see Appendix F) 
than the older half of the faculty (40 or above). Consequently, the 
third null hypothesis is accepted. 

Tenure . Chi-square contingency tests were applied to tenured (N = 34) 
and non tenured (N = 33) faculty members' responses to the faculty 
attitude survey questionnaire (see Appendix G). Again, only one question 
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(number 12) showed significant differences - (12) I'm not sure the 
administration would be able to view this information in the proper 
perspective (P = .05). Consequently the fourth null hypothesis is 
accepted. 

Years at Austin College . Faculty respondents when divided in half 
by number of years at Austin College (5 years or less and six or more) 
resulted in significant differences on three questions from the faculty 
attitude survey questionnaire (see Appendix H). Those at Austin College 
longer agreed more with the statement (2) I find this information very 
informative (P = «05)o However , those in residence longer, disagreed 
more with the statements (12) I'm not sure the administration would be 
able to view this information in the proper perspective (P = .01), and 
(18) It's not really clear what the college expects of me as a teacher 
(P = .05). Since one of the significant differences involves a question 
(number 2) designed to deal with the perceived value of the data, the 
fifth null hypothesis cannot be fully .accepted although most of the data 
indicates perceived value of the data. Also threat (question number 12) 
seems to be greater and role clarity (question number 18) less for 
faculty with fewer years at the college. These results also suggest 
that a clear, full acceptance of the fifth null hypothesis is con- 
traindicated, but the vast amount of the data calls for at least partial 
cbnfirmatiohir ^ • ••. 

Rank. Chi-square contingency tests were conducted of faculty response 
to the attitude survey questionnaire with the faculty divided into two 
catagories by ranks (1) Instructor or Assistant Professor, and (2) Associate 
Professor or Full Professor. No statistically significan differences (see 
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Appendix I) were found so the sixth null hypothesis is confirmed or 
accepted . 

Area * When faculty is divided by area of academic discipline 
(Humanities, Social Science, or Physical Science) two questions attained 
statistical significance (see Appendix J). Humanities faculty members 
were more likely to disagree with the question (7) I was surprised that 
I didn't get more positive recommendations (P - .05) but more likely to 
agree with (10) I wouldn't mind if this information were made public 
(P = .05). Neither of the statistically significant questions indicate 
the data is perceived as not being valuable, but Humanities faculty 
seem to be less threatened by the data and the prospect of making it 
public. Several other questions were designed to ascertain threat but 
no significant differences were found. Consequently, the seventh null 
hypothesis is mostly confirmed save for the one question indicated 
above. Chances of accepting the null hypothesis when indeed it is false 
seem quite small indeed. 

Choice Quar tiles . The faculty was divided into four catagories acc- 
ording to whether they fell in the top or bottom half of the professor 
most recommended list and of the professor least recommended list. 

(H-H or high most recommended and high least recommended, H-L or high 
most and low least, L-H or low most and high least, L-L or low most and 
low least). Significant differences were found on Chi-square contingency 
tests for four questions on the faculty attitude survey questionnaire - 
(see Appendix K)|. Those in the L-H catagory tended to agree more that 
(1) I don't care who it is, it makes me nervous knowing that anyone has 
information like this, (P = .05), and to disagree more that (8) I would 
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like to get confidential information like this every year, that (10) I 
wouldn't mind if this information were made public and that (18) It's not 
really clear what the college expects of me as a teacher. The eighth 
null hypothesis is rejected. Those receiving a low number of positive 
recommendations and a high number of negative recommendations are 
clearly more threatened than those falling in more enhancing choice 
catagories. 

Highest Degree Attained . When faculty were divided by highest attined 
degree (B. A. - M. A. vs Doctorate) no significant differences were found 
on the faculty attitude survey questionnaire (see Appendix L). Con- 
sequently, the ninth and final null hypothesis is accepted without 
reservation. 
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DISCUSSION AND CONCLUSIONS 

In general, all the results of this study revealed "sensible" or 
predictable results. The greatest gain of the study was the creation 
of some codes as to why students choose to most recommend or least 
recommend professors and courses (see Appendix B). The fact that the 
first null hypothesis of no difference in faculty attitude responses 
between those receiving coded and verbatim interview responses was 
clearly accepted indicates that the codes employed will be as effective 
as interview results when used as a phrase checklist next year on a 
revised sociometric questionnaire. 

The procedure also seems to have been largely well received by the 
faculty as indicated by the results of the measure of central tendency 
and the qualitative evidence alluded to in the previous section of this 
report. The mean results indicated a general lack of defensiveness that 
is probably attributed to the fact that the procedure used only gave 
individual professors the sociometric and interview results. Nothing 
was given to the administration so that there was no chance of the data 
having any influence on status, reputation, or salaries. No control 
group was employed to test this notion but questions on making the data 
public or available to the administration were not received with 
agreement by many faculty members. Even the presence of such a control 
group would have probably affected the results of the experimental group. 
This data is sensitive and potentially volatile. The faculty responses 
might not have indicated either low perceived threat or high perceived 
value. Rather Festinger's (1956) dissonance theory probably would have 
operated where faculty would have either ignored or distorted the 
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results. This probably explains why four of the six professors who refused 
to respond to the faculty attitude survey questionnaire were chosen as 
low on the professor most recommended list. 

Age difference (39 or below and 40 or above) displayed only one 
statistically significant result in that younger faculty members seemed 
more concerned that students would recommend easy courses and professors. 

It seems a matter of pride more perhaps, that the younger professor proves 
he is no pushover. It seems related that those with fewer years in residence 
at Austin College (5 or less vs 6 or more) registered a lesser degree of 
role clarity (question 18), more mistrust in the administration ability 
to view this information in the proper perspective (question 12), and that 
they found the information significantly less informative (question 2). 

These questions could indicate that role definition by students is placed 
in secondary position until role clarity problems from the administration's 
point of view is first resolved. Until that resolution, student data 
might be viewed as threatening however, this resolution seems to occur 
before it can be detected as differences by academic rank (instructor- 
assistant professor vs associate professor-full professor). 

Quite naturally, those faculty members who were not exactly enhanced 
by the sociometric choices or the interview results were less willing to 
have the information made public (question 10), more nervous that anyone 
has data of this nature (question 1), and less eager to want such 
information every year (question 10). However, the availability and 
communication of such information even the knowledge that it exists, 
tends to create dissonance or at least activate some sort of cognitive 
tension system which demands some sort of response even if the information 
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is distorted or responded to with selective inattention* But even these 
responses require the expenditure of energy. However there is some 
evidence that change is produced utilizing this instrument and procedure. 
Perhaps change will be an increasingly likely response when next year's 
results are compared with the results outlined in this report* This 
comparision will be made each year until a four year cycle is established. 
When the point is reached where a faculty member can compare a full four 
year student generation, it seems redundant to include earlier responses 
since change surely must occur on a more regular basis* 

Future development of this method and procedure seems to be Strongly 
indicated from the results of this study. 

Curiosity dictates that revised instruments should be developed for 
use in public schools where objectives are often anything but institutions 
which offer self-paced learning and encouragement toward creativity - 
they socialize instead (sometimes unsuccessfully). However, disciplined 
reason dictates that the procedure be taken to a logical developmental 
conclusion or to some form of closure at the higher educational level 
before attempting to move to another level* Consequently, it is 
recommended that larger institutions should be investigated for the 
development of efficient data collection if nothing else. Junior colleges 
present special problems because their student generation length is only 
two years, and more often than not the majority of students live off 
campus. Junior colleges also have the advantage of low peer influence 
because of these residential differences. In any event, this approach 
seems to show promise as a useful method to bring abbut educational 
reform, and it is recommended that further research eventually be 
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undertaken at all levels of education— higher , secondary, and primary. 
It would also be interesting to do some cross cultural research. 
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APPENDIX A 

COMPONENTS OF EFFECTIVE TEACHING AS PERCEIVED BY STUDENTS 



Factor 

coeffecient 



SCALE 1. ANALYTIC /SYNTHETIC APPROACH 

1. Discusses points of view other than his own #70 

2. Contrasts implications of various theories .66 

3. Discusses recent developments in the field .64 

4. Presents origins of ideas and concepts .60 

5. Gives references for more interesting and involved points .53 

6. Presents facts and concepts from related fields .53 

7. Emphasizes conceptual understanding .46 



SCALE 2. ORGANIZATION/CLARITY 



8. Explains clearly .78 

9. Is well prepared .63 

10. Gives lectures that are easy to outline .62 

11. Is careful and precise in answering questions .61 

12. Summarizes major points .51 

13. States objectives for each class session .50 

14. Identifies what he considers important .47 



SCALE 3. INSTRUCTOR-GROUP INTERACTION 



15. Encourages class discussion .70 

16. Invites students to share their knowledge and experiences .65 

17. Clarifies thinking by identifying reasons for questions .64 

18. Invites criticism of his own ideas .62 

19. Knows if the class is understanding him or not .58 

20. Knows when students are bored or confused .57 

21. Has interest and concern in the quality of his teaching .48 

22. Has students apply concepts to demonstrate understanding .43 



SCALE 4. INSTRUCTOR- INDIVIDUAL STUDENT INTERACTION 



23. Has a genuine interest in students .74 

24. Is friendly toward students .71 

25. Relates to students as individuals .69 

26. Recognizes and greets students out of class .68 

27. Is accessable to students out of class .65 

28. Is valued for advice not directly related to the course .64 

29. Respects students as persons .60 
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APPENDIX A (cont.) 



Factor 
coef fecient 



SCALE 5. DYNAMISM/ENTHUSIASM 



30. Is a dynamic and energetic person .80 

31. Has an interesting style of presentation .76 

32. Seems to enjoy teaching .74 

33. Is enthusiastic about his subject .65 

34. Seems to have self-confidence .64 

35. Varies the speed and tone of his voice .63 

36. Has a sense of humor .53 



Based on the 1968 survey - N = 1015 
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^ STUDENT SOCIOMETRIC SURVEY 

INTRODUCTION: This questionnaire is part of a sociology research project. Please do not 

sign your name anywhere on this questionnaire . Your real opinions and attitudes are 
wanted. Your questionnaire will be numbered so that a random sample may be drawn later 
for some student-run interviews, but no faculty member wil I have access to the list of 
names the numbers stand for, and students working on the project will only have access 
to information on a strict "need to know" basis. Your effort and cooperation is very 
much appreciated. 



STATISTICAL DATA : Circle the number of the appropriate response. Please answer a I I 

statistical questions . 



1. Fraternity or sorority member? 


3. 


Your dormitory? 




(1) Yes (2) No 




(1) Baker 


(4) 


Clyce (7) Thompson 


If YES, which organization? 




(2) Dean 


(5) 


Caruth (8) Apts. 


2. What is your classification? 
(1) Freshman (3) Junior 


4. 


(3) Luckett 
Your major? 


(6) 


Coffin 


(2) Sophomore (4) Senior 


5. 


Your sex? Male 


Female 



I NSTRUCT I ONS : Please list as many as three (3) students, professors, and courses 

according to the questions below. You need not use all three choices, but any more 
than three cannot be tabulated. 

I. List the three (3) students whose opinions and attitudes you respect the most 
concerning various aspects of campus affairs. 

(I) (2) (3) 



2 . 



3. 



4 . 



What three (3) professors that you know would you be most I ikely to recommend 
to other students; what three (3) professors would you be least likely to 
recommend? 

(MOST) (LEAST) 



(1) 








( 1 ) 




(2) 








(2) 




(3) 








(3) 




What 
(1 ) 


three (3) 


courses wou 1 d 


you 


be most 1 i kely to 


recommend to other students? 


(2) 


Dept. 


& Course # 




Name of Course 


1 nstructor 


(3) 


Dept. 


& Course if 




Name of Course 


1 nstructor 




Dept. 


& Course # 




Name of Course 


Instructor 


What 

(1) 


three (3) 


courses would 


you 


be least likely to 


recommend to other students 


(2) 


Depf. 


& Course if 




Name of Course 


1 nstructor 




Dept. 


& Course # 




Name of Course 


1 nstructor 



( 3 ) 



Dept. & Course ft 



Name of Course 



I nstructor 



APPENDIX C 



SOCIOMETRIC DATA 



CODE NUMBER 2229 



For the past several years, a sociometric questionnaire has been under development 
which might provide faculty members with some indication of what students think 
of them and their courses. The data has been carefully handled so that no one 
student would have access to any more than a small portion of the information. 

For the first time last year, upon request, faculty members were given the data 
which concerned their courses. These faculty members responded to a questionnaire 
concerning the student reaction. 

The information that follows indicates the number of times you were mentioned by 
students in response to four questions: (1) "...professors you would most 

likely recommend to other students," (2) ".. .professors. . .least likely to 

recommend," (3) "...courses you would most likely recommend to other students," 
and (4) ".. .courses. . .least likely to recommend." 

most likely least likely 

recommended recommended 

no. choices no. choices 



1. Professor choices (questions 1 & 2) 58 2 

2. Course choices (questions 3 & 4) 

Basketweaving 11... 13 0 

Basketweaving 26... 4 14 

Advanced Basketweaving 88... 1 31 



The pages that follow will provide some idea of your standing relative to other 
professors and courses. Please exercise caution . No correction factor has been 
applied to account for choice differences due to the number of students to which 
faculty are exposed, number of years at the college, etc. In fact, many have 
felt that such a correction factor would be impossible to calculate. There are 
some desirable aspects to the fact that a direct comparison of professor and 
courses cannot be made. This information is primarily for use by individual 
faculty members who wish to acquire more information to use in self evaluation. 
In other words, the intent of this research is to serve the college community by 
serving the faculty rather than the students or administration. 

For the first time, a portion of the students have been interviewed as to why 
they have made the choices they made. This information is provided for you 
immediately following the computer print out. 

Finally, a questionnaire is included for your reaction to the whole experience. 
Your cooperation in completing this short questionnaire is very much appreciated. 
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CODE NUMBER 2309 



EXAMPLE OF VERBATIM RESPONSE 



I 

| 

ERIC 



POSITIVE 

Is just an extrordinary man and would suggest that someone take his 
course just to listen and watch him work. He knows that subject better 
than any other man. 

He likes students. Will BS during class and outside of. Not a technican 
but authetic scientist. 

Since he's been around since dinosaurs, knows it all well, obviously; 
wealth of information; he understands biology; just being with him 
enlightening; helps students understand — he stays with you until you do 
(or gives up because you're such a knucklehead — I can't stand it (giggle!)). 
He always says it isn't hard--attitude great--because he laughs about it 
and the fact that you can co well; still got a heck of a lot of energy, 
so school should not put him out--it will destroy him because he's still 
got so much in him to give. 

The only reason I put him down was that he was my advisor. He is kind. 
Always listens, glad to talk to me. I've never had a course with him. 

I just got to be good friends to where I could talk with him. 

His knowledge and experience is wide based, he cares for the student, 
he doesn't take his own subject overly too seriously, he's so flexible-- 
he'll let you go off on a tangent in class, he's still trying different 
methods even though he's been teaching for so long, he seems to enjoy 
his students' success, fantastic professor. 

A very organized person, in style of presenting material. He explains 
in detail all ideas. 

Have him for Biology--very impressed with his lack of structure. He 
realizes that he has never taught this course before, but trys things 
and changes when they fall through and trys something else. Very liberal 
teacher even though he is a conservative man (this is understandable 
being that he is 64 years old). His main concern is exercises in a 
capacity to learn, but he is not forceful. He makes a very valid attempt 
to understand when people do not come up to his expectations. If there 
is a personal problem that is keeping them down, keeping the student 
from exercising his capacity, he tries to help you get through it. It 
all boils down to the fact that he understands that there are personal 
problems involved that keep a student from doing his best in school, and 
that is to be accounted for. I really appreciate the life he instills 
in things--really enjoy what he is doing. Trys to work things so that 
people really can enjoy it. Very good at making things understandable 
that don't seem practical , that otherwise seem meaningless. Guesto-- 
great pains to show that concepts really are important, meaningful and 
applicable. 
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CODE NUMBER 2309 cont. EXAMPLE OF VERBATIM RESPONSE (cont.) 

He knows everything but is patient in explaining and he always answers 
questions. He allows questions during his lectures. 

Goes out of his way to give all possible information on questions; 
interested in you as a person, as well as student; open-minded (quite a 
quality); expects a lot out of you--lectures well planned --out lined, 
easy to follow. 



NEGATIVE 

He is very nice as a person but he fails to understand your course 
problems and he doesn't explain in a way to make you understand either. 

To strait forward--cut and dry in class — no personality— too factual. 
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PROFESSOR MOST CHOICES 



Ml 

M2 

M3 

M4 

M5 

M6 

M7 

M8 

M9 

M10 

Mil 

M12 

M13 

M14 

M15 

M16 

M17 

M18 

M19 

M20 

M21 
M2 2 
M2 3 
M2 4 
M25 

M26 

M27 

M28 

M29 

M30 

M31 



Total 

Coll- 

ege 

Presentations indicate material thoroughly understood — 



always well prepared 183 

Good rapport - relates well to students; establishes 

meaningful relationships with students 181 

Has own style or educational approach - different, 

interesting, enjoyable 136 

Patient - takes extra time and effort to help students J 15 

Personality adds to course - good attitude 107 

Takes interest in student development 96 

Flexible - takes your interests and objectives into 
account 79 

Relaxed hospitable environment for learning - not 

like a factory assembly line 72 

Encourages input from students - student involvement.. 54 
Tolerates disagreement - never cuts you down for your 

opinions and ideas 35 

Values high productivity - challenges class 33 

Provides inspiration to student 32 

Enjoys profession; takes it seriously 31 

Puts contemporary problems into perspective - focuses 

on real concern of students 31 

Interested in subject he is teaching 29 

Empathic - aware of student problems 26 

Reasonable tests - optional find, fair grades, no 

surprises, etc 26 

Low pressure level - expectations of students 

realistic 23 

A scholar - concerned with the understanding of truth 23 
Discussion oriented - keeps discussion going but 

doesn't dominate 23 

Allows personal freedom - freedom of expression 22 

Knowledgeable in more than one area 20 

Tells it like it is - doesn't beat around bush 18 

Stimulates students - provokes thought 17 

In command of course - knows what he's doing, where 

going, and how to get there 16 

Makes expectations of students clear and precise 15 

Admirable person 13 

Well informed - "in tune" to college and community 

activities 12 

A good listener - really tries to understand students.. 12 

Opens up to people 11 

Helpful - gives good advice 10 



Total 

Code 

# z zii 

n 



% 



6 



4 



I 



( 
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Professors Most Choices (continued) -2- 

Total 

Coll- 

ege 



M32 Doesn’t try to be an authority figure 9 

M33 Concern for practicable application 'ctf facts 9 

M34 Course well paced - not too fast; not too slow 8 

M35 Has a good reputation on campus with students and 

professors 8 

M36 Appearance pleasing to others 7 

M37 Treats generalizations so they can be microscoped 

into specifics 

M38 Initial experience provided a ;ood foundation for 

further study 6 

M3 9 Covers a variety of topics in course 6 

M40 Never humiliates students in class 5 

M41 Unprejudiced - accepts student appearance, dress, 

life style, etc 5 

M42 Makes the abstract relevant 4 

M43 Methodical 

M44 Made a contribution towards choosing a major field 

of concentration 2 

M45 Reasonable attendance policy - non-compulsory, etc. 2 

M46 Creative - original ideas, appearance, etc 2 

M47 Is a member of an ethnic group 1 

M48 Hands out teacher evaluation form 1 

M49 Not stuck on himself 1 
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Total 

Code 

<tZZZ<1 
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LI 

L2 

L3 



L4 

L5 

L6 

L7 

L8 

L9 

L10 

LI 1 
L12 
L13 
L14 

L15 



L16 

L17 

L18 

L19 

L20 

L21 

L22 

L23 



PROFESSOR LEAST CHOICES 



No diversity in lectures — not pertinent, boring— 

Puts you to sleep 

Generally lousy and/or irrelevant- -do not like teaching 

style — have not learned anything 

Tests unreasonable-ambiguous or material not 
covered or opinions (no right answer) or picky 

detail wrote, memorize 

Rigid — closed minded — dogmatic at times 

Not really interested in students— phoney-always 
too busy 

Always cuts you down— in class— on papers, etc., 

stifles 

Conceited with an air of superiority. 

Sometimes on an ego trip- -you have to walk on egg 

shells — so fascinated with self 

Expects too much — goes too fast—assumes you know 

what is going on — and won't take time to explain 

Antagonistic toward students — values, opinions, 

attitudes, long hair, dress, etc.— stifles 

Never had him, but my friends don't like him 

Course disorganized and/or unprepared 

Grading picky — too specific, too hard, no flexibility.. 
Seems uninterested in students — material — maybe 

teaching itself 

Humor or lanquage offensive— either corny, or off- 
color, profane, anti-feminist, or something, 

irritating 

Completely unable to communicate with him (her) 

Defensive, insecure, or nervous around students, 

easy to shake up or get him off subject 

Most all professors irritating in one way or 

another. This one has a lot more than his share 

Poor discussion leader, disorganized or unprepared 
or dominates or suppresses, or forbids all together.... 
Plays favorites — unfair to non-majors or some other 

student category 

Too often can't answer questions he should know 
seems unqualified to teach at Austin College- 

Uninformed 

Rambles — gets off subject on tangents 

So ambiguous you never know what to expect. His 
approach does not make you learn 



Total 


Total 


Coll- 


Code 


ege 




97 .. . 


..7 


.74 . . . 




43 .. . 




39 .. . 




36 ... 




34 . . . 




33 .. . 




31 . . . 




31 . . . 




29 . . . 




28 . . . 




27 . . . 




27 . . . 




26 . . . 




26 ... 




26 ... 




22 . . . 


..1 


21 ... 




21 . . . 




21 . . . 




20 ... 




19 . . . 




14 ... 
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Professor Least Choices (continued) -2- 

Total Total 

Coll- Code 

ege ft Z * 2 - 7 ^ 



L24 Have no respect for him as professor or person 

despicable 14 

L25 No complaints allowed — must agree with what is said 

or else 13 

L26 No depth — skims material and does not explain as 

should 11 

L27 Doesn't generate interest — you have to already be 

interested in the material 10 

L28 Dead Lecturer — can't communicate — detached or 

impersonal — "this is a recording" 8 

L29 Never know where you stand — too few tests or 

indicators 8 

L30 Inexperienced and/or immature — does not know his 

own strengths and weaknesses 8 

L31 Quite ungraceful on own mistakes — never admits he 

is wrong 7 

L32 Forces students to learn for themselves — if anything 

is learned 7 

L33 Didn't like texts and/or assigned readings 5 

L34 Patronizing — like parent talking to child — talks 

down to you 5 

L35 Attendance policy unreasonable or unfair 5 

L36 Preaches — does not lecture — hung up with 

religion 4 

L37 Says a lot he can't back up — confuses students 4 

L38 Course didn't cover material it was supposed to 

cover 4 

L39 Was my advisor — really fouled me up 3 

L40 Distasteful personal appearance or image 2 
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COURSE MOST CHOICES 



r 



Total 

Col- 

leqe 



MA Material itself compelling - challenging - 

worthwhile, interesting 194 

MB Professor competent teacher - likable - makes the 
course - doesn't go too fast - explains till you 

get it 

MC Practical - applies subject matter to relevant 

situations and problems 97 

MD Good course - really enjoyed it 60 

ME Good reading material. 47 

MF Liked the way course was structured or organized. .45 
MG Provides opportunity to work on things you choose 

or initiate 

MH Professor helped by making material more 

interesting and/or understandable 30 

MI Stimulating lectures - well presented 30 

MJ Makes you think and/or reevaluate your ideas 

about the subject matter 29 

MK Lots of two-way interaction between students 

and professor. 26 

ML Different - represents an interesting change of 

pace for me 26 

MM Tests and/or final exam good learning ex- 
perience 21 

MN Projects good - gets you into the real world 16 

MO Multidisciplinary - covers related material from 

other fields - learning transfer to other courses. . . .15 

MP You work hard but get something out of it 14 

MQ Good reputation - never taken the course 13 

MR Professor keeps up with developments in his 

field c 



MS Professor doesn't try to sell a point of view - his 

impartiality lets you make decisions on your own. . 4 



Total 

Code 



*2 2J 1 



% 



(0 
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COURSE LEAST CHOICES 



Total 

Col- 

lege 



I 



j 




LA I dislike this course because of the Professor .... 66 
LB Course was a "blow-off" - too simple; too 

much review, etc 58 

LC Boring course - dull subject; no connection 

between lectues and text 51 

LD Not a good course - not the course described in 

catalogue; too much busy work 34 

LE Course lacks any practicable application 30 

LF Did not like text and/or choice of literature - 

outdated 30 

LG Material presented in uninteresting manner.... 29 

LH Class too large - almost no student, teacher.. 

interaction 28 

LI Course needs redesigning - almost no organi- 
zation, planning, or structure 27 

LJ Poor tests - too long; too much emphasis on 

pop tests or correct answers; covers too much 

material; too much trivia, etc 24 

LK Too much is expected of me in too short a 

time 22 

LL This course should not be required 21 

LM Professor lacks interest in course; no enthusiasm, 

bad attitude 19 

LN Professor doesn't communicate or explain 

subject well 19 

LO Not enough emphasis on the fundamentals of 

course 18 

LP Too much pressure - standards too high; course 

requires too much time 17 

LQ Course is hard - professor needs to be more 

helpful and understanding of students J6 

LR This course is designed for those majoring in 
the subject - another course for non-majors is 

needed 13 

LS I didn't learn anything - just memorize; can't 

remember anything 13 

LT Material too detailed - over students' heads .. 12 

LU Course offered too early; meets too often; class 

too long 12 

LV Professor doesn't like to help students having 

trouble - doesn't make himself available for 

help 11 

LW Course doesn't demand enough of students .... 10 
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Course Least Choices (continued) 



LX 



IY 

LZ 

LZA 

LZB 

LZC 

I, ZD 
LZE 
LZF 

LZE 

LZF 

LZG 



Total 

Col- 

es^ 



Total 

Code 






Professor expects students to have a good 
background in subject - I'm not interested 

in subject 9 

Professor indicates that he does not thoroughly 

understand material he is teaching 8 

Professor cuts you down - bad atmosphere 8 

Professor pushes his opinion upon students 8 

Professor and course have bad reputation on 

campus 8 

This course will not enhance the student's 

personal development 8 

Professor hard to relate to - stuck on himself .. .7 
Too abstract - could not understand the subject. .6 
Professor does not clearly state what he expects 

from students 5 

A pre-requisite should be required to take this 

course 3 

Professor not up-to-date on latest developments 

in field 2 

Professor lacks confidence in himself 2 
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Code # 



FACULTY EVALUATION FORM 



I. Introduction : This questionnaire is designed to assist you in registering 
your reactions to the student opinions as noted in the materials provided 
in the first sections of this packet. Please do not sign your name any- 
where on this form - your code number is listed above so that follow-up 
requests for completion can be made and so that no one handling this 
questionnaire will know whose responses they are tabulating. Only 

the experimentor will have access tc the code and his behavior is 
regulated by strict legal and ethical codes which forbid him from compro- 
mising sensitive data to the possible detriment or discomfort of those 
providing the data . 

II. Instructions: Please respond to the following statements by circling that 
alternative which most closely approximates your feelings about the matter 
in question SD (Strongly Disagree) . D (Disagree). ? (Uncertain. Don't 
Know or No Opinion), A ( Agree). and SA (Strongly Agree). 

1. I don't care who it is, it makes me nervous knowing 

anyone has information like this 

2. I find this information very informative 

3. I didn't learn anything about myself or my courses 

from this that I didn't already know 

4. This information helps me evaluate myself 

5. This data taking should be stopped 

6. I was surprised that I didn't get more negative 

recommendations 

7 . I was surprised that I didn't get more positive 

recommendations 

8 . I would like to get confidential information like this 

every year 

9. I don't understand the computer printouts 

10. 1 wouldn't mind if this information were made public. 

11. 1 think I can improve my teaching due to my 

receiving this information 

12. I'm not sure the administration would be able to view 

this information in the proper perspective 

13. Student opinions are important to me 

14.1 wouldn't rnirid if the administration usdd this 

information as a part of their evaluation of my 
performance or my effectiveness as a teacher- 

15. 1 don't believe this information reflects the 

opinions and attitudes of students majoring in 
my discipline 

16. Student comments make the student recommendations 
of me and my courses more understandable 

17. 1 suspect students recommend easy courses and 

professors 



SD 


D 


? 


A 


SA 


SD 


D 


? 


A 


SA 


SD 


D 


7 


A 


SA 


SD 


D 


? 


A 


SA 


SD 


D 


? 


A 


SA 


SD 


D 


7 


A 


SA 


SD 


D 


7 


A 


SA 


SD 


D 


? 


A 


SA 


SD 


D 


? 


A 


SA 


SD 


D 


7 


A 


SA 


SD 


D 


7 


A 


SA 


SD 


D 


? 


A 


SA 


SD 


D 


7 


A 


SA 


SD 


D 


? 


A 


SA 


SD 


D 


7 


A 


SA 


SD 


D 


7 


A 


SA 


SD 


D 


? 


A 


SA 
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18. 


It's not really clear what the college expects 

of me as a teacher 


SD 


D 


? 


A 


SA 


19. 


I find this whole experience degrading 


SD 


D 


? 


A 


SA 


20. 


No matter how difficult and complicated, a correction 
factor should be calculated so that all scores for 
professors and courses could be compared 


, SD 


D 


9 

• 


A 


SA 


21. 


Wouldn't really be interested in more precise 
information Just to be better able to compare 
myself with others 


, SD 


D 


? 


A 


SA 


22. 


The choice scores and student comments do not 
provide enough basis for any kind of corrective 
action on my part 


. SD 


D 


? 


A 


SA 


23. 


The student comments constitute the best information 
in the whole ''package" 


SD 


D 


? 


A 


SA 


24. 


Student comments are useless to me 


SD 


D 


? 


A 


SA 



III. Please check the alternative below which most nearly reflect your 
feelings about the value of student opinion: 

1 . Completely worthless - students are not qualified to adequately judge 

such matters - their opinion is insufficient and unnecessary for pro- 
fessor evaluation. 

2 . Almost Completely Worthless 

3. Mostly Worthless 

4. Of some value but very little compared with other factors 

5. Worth about equal the consideration given other factors - necessary 

but insufficient for valid professor evaluation 

6 . Of some value - probably more so than any other factors 

7. Quite valuable 

8. Almost completely valuable and relevant 

9. Completely valuable and relevant in the evaluation of professors - 

without student opinion, other factors don't matter. It is necessary 
and sufficient for evaluation. 

IV. Your comments are earnestly solicited. (Please Use Back of Sheet If 

Necessary) 
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APPENDIX D 

RAW DATA AND CHI-SQUARE CONTINGENCY TESTS FOR FACULTY QUESTIONNAIRE 
RESPONSES FOR FACULTY RECEIVING CODED STUDENT INTERVIEW RESPONSES 
VS FACULTY RECEIVING VERBATIM INTERVIEW 



S's 1 


CM 

Q 

CO 


D 3 


? 4 


A 5 


SA 6 


NR 7 


SD&D 8 


9 

?&NR 


SA&A 10 


Chi 2 


P 11 


1. Verbatim 


10 


14 


5 


2 


1 


4 


24 


9 


3 






Coded 


6 


16 


4 


2 


1 


2 


22 


6 


3 


0.316* 


Not Sig. 


2. Verbatim 


2 


4 


7 


14 


6 


3 


6 


10 


20 






Coded 


3 


5 


3 


17 


0 


3 


8 


6 


17 


1.162 


Not Sig. 


3. Verbatim 


1 


17 


4 


9 


2 


3 


18 


7 


11 






Coded 


0 


17 


3 


5 


3 


3 


17 


6 


8 


0.207 


Not Sig. 


4. Verbatim 


1 


• 4 


5 


18 


5 


3 


5 


8 


23 


ft 




Coded 


2 


2 


4 


16 


8 


4 


4 


8 


19 


0.120 


Not Sig. 


5. Verbatim 


10 


15 


7 


0 


1 


3 


25 


10 


1 


ft 




Coded 


5 


15 


6 


1 


1 


3 


20 


9 


2 


0.572 


Not Sig. 


6. Verbatim 


0 


11 


12 


7 


3 


3 


11 


15 


10 






Coded 


1 


12 


10 


4 


0 


4 


13 


14 


4 


2.413 


Not Sig. 


7. Verbatim 


3 


16 


11 


3 


0 


3 


19 


14 


3 


ft 




Coded 


0 


11 


11 


5 


0 


4 


11 


15 


5 


2.308 


Not Sig. 


8. Verbatim 


2 


2 


9 


16 


5 


3 


3 


12 


21 


ft 




Coded 


1 


4 


4 


13 


5 


3 


6 


7 


18 


2.186 


Not Sig. 


9. Verbatim 


6 


23 


1 


2 


1 


3 


29 


4 


3 


ft 




Coded 


7 


17 


0 


2 


1 


4 


24 


4 


3 


0.099 


Not Sig. 


10. Verbatim 


8 


7 


5 


12 


1 


3 


15 


8 


13 


ft 




Coded 


5 


9 


5 


7 


1 


4 


14 


9 


8 


0.916 


Not Sig. 


11. Verbatim 


2 


7 


7 


14 


2 


4 


9 


11 


16 






Coded 


3 


3 


9 


12 


1 


3 


5 


12 


13 


0.584 


Not Sig. 


12. Verbatim 


2 


9 


5 


15 


1 


3 


12 


8 


16 






Coded 


3 


10 


5 


7 


4 


3 


6 


12 


8 


0.556 


Not Sig. 


13. Verbatim 


0 


0 


1 


20 


12 


3 


0 


4 


32 


ft 




Coded 


0 


1 


0 


16 


11 


3 


1 


3 


27 


1.200 


Not Sig. 


^Subject s 








5. 

Agree 








9 


i 

Undecided 


& No Response 



"Strongly Disagree 

L , 

Disagree 

Undecided or No Opinion 



8 



Strongly Agree 
^No Response or Refused 
Strongly Disagree & Disagree 



10 

11 



Agree & Strongly Agree 

Probability or Level of 
Significance 



Eventhough cells were combined, one or more expected frequency cells were less 
than 5.0. 
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APPENDIX D (cont.) 



S's 1 


SD 2 


D 3 


? * 


A 5 


SA 6 


NR 7 


SD&D 8 


9 

?&NR 


SA&A 10 


2 

Chi 


P 11 


14. Verbatim 


4 


7 


8 


11 


3 


3 


11 


11 


14 






Coded 


4 


1 


4 


13 


6 


3 


5 


7 


19 


3.543 


Not Sig. 


15. Verbatim 


3 


6 


16 


7 


1 


3 


9 


19 


8 






. Coded 


3 


10 


7 


5 


2 


4 


10 


14 


7 


0.507 


Not Sig. 


16. Verbatim 


0 


3 


8 


18 


4 


3 


3 


11 


22 






Coded 


1 


4 


7 


11 


2 


6 


5 


13 


13 


2.622 


Not Sig. 


17. Verbatim 


4 


17 


9 


3 


0 


3 


21 


12 


3 


"ic 




Coded 


0 


13 


9 


5 


1 


3 


13 


12 


6 


2.523 


Not Sig. 


18. Verbatim 


4 


20 


0 


6 


3 


3 


24 


3 


9 






Coded 


2 


14 


4 


7 


1 


3 


16 


7 


8 


2.902* 


Not Sig. 


19. Verbatim 


9 


16 


6 


2 


0 


3 


25 


9 


2 






Coded 


5 


17 


5 


0 


1 


3 


22 


8 


1 


0.212 


Not Sig. 


20. Verbatim 


6 


6 


14 


4 


2 


4 


12 


18 


6 


k 




Coded 


0 


8 


11 


7 


1 


4 


8 


15 


8 


0.991 


Not Sig. 


21. Verbatim 


0 


7 


6 


17 


3 


3 


7 


9 


20 






Coded 


2 


8 


8 


5 


4 


4 


10 


12 


9 


4.784 


Not Sig. 


22. Verbatim 


1 


12 


8 


11 


1 


3 


13 


11 


12 






Coded 


0 


8 


6 


9 


4 


4 


8 


10 


13 


0.190 


Not Sig. 


23. Verbatim 


0 


3 


9 


13 


8 


3 


3 


12 


21 






Coded 


1 


6 


9 


10 


1 


4 


7 


13 


11 


4.416 


Not Sig. 


24 i Verbatim 


14 


17 


1 


1 


0 


3 


31 


4 


1 






Coded 


8 


17 


1 


0 


0 


4 


25 


6 


0 


1.679 


Not Sig. 



Subjects 

l 

Strongly Disagree 
Disagree 



Agree 

^Strongly Agree 
No Response or Refused 



10 

11 



8 



Undecided or No Opinion Strongly Disagree & Disagree 



Undecided & No Response 

Agree & Strongly Agree 

Probability or Level of 
Significance 



% 

Eventhough cells were combined, one or more expected frequency cells were less 
than 5.0. 
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APPENDIX E 



Code Number and Comments 

2201 - Since there was only one response on my teaching (quite under- 
standable since I normally don't teach courses), I am unable to answer 
many of the questions. 

2308 - No comments. 

2221 - I was flattered but baffled by the kind comments students made. 

2205 - Certain aspects need more control to make study more valid; 
case in point: 20 students off campus when the data was collected 
had the best possibility of evaluating me. Hope you continue and 
refine the instrument. 

2204 - I would like to see responses broken down by concentrators and 
non-concentrator s • I feel that at A. C. with its open and innovative 
academic program, this seems as a useful tool. We should be "objective" 
about ourselves and our professional performance to the extent that 
broader use of the instrument should be made. I have not considered 
the Birkman analysis an invasion of my privacy — it was widely shared. 

By the same token, J. do not consider this analysis an invasion of my 
classroom "privacy,"— -and feel that these results could be shared for 
mutual benefit. I checked number 5 in the alternatives as the one 
that most described my feeling. It was not too close. I feel that the 
instrument is quite valuable, but that other measurements need to be 
taken to round out the picture. (The patient is not always the best 
doctor in analysis.) I hope you continue the analysis and continue 
to refine it. 

2203 - I am a perfectionist and am inclined to have a poor self-image. 

This evaluation really gave me an ego massage. Apparently, I'm a lot 
more successful than I had thought. I think I will be an even better 
teacher next year because the negative side of this evaluation will 
help me to improve my weak points. I appreciate you making this 
evaluation possible. I represents a lot of time and hard work. Thank 
you. 

2202 - We are given ratings and comments at both ends of the spectrum 
(most popular-least popular) but I would like to hear from students who 
are not committed in either direction. Ideally, I would like the 
evaluation of all of my students to be included. 

2301 - The computer print-out fails to give much meaningful information, 
and could be rather easily misinterpreted. Since you have gone to the 
trouble of listing professors in descending order of choice (both positive 
and negative) rather than by code number sequence one would assume that 
you intended to point out who was the most or least recommended by the 
students. Even a cursory treatment of the raw data would be more informative 
and could easily be done on the computer with the information already 
coded. I suggest that a ratio of positive to negative responses reduced 
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APPENDIX E (cont.) 



Code Number and Comments 

2301 (cont.) - to percentages would yeild a better measurement of a 
faculty member's standing with students than raw data as you have projected 
it. Applying this method to your data the following few examples 
demonstrate my point (values are estimated). 

Your rank ordering Percentage Positive 

positive Code ID Responses 



1 


2138 


727. 


2 


2221 


967. 


3 


2213 


957. 


4 


2217 


797. 


5 


2242 


907. 


38 


2111 


957. 


39 


2108 


207. 


40 


2304 


947. 


41 


2208 


937. 



With a little more effort you might check the possibilities of sexist 
bias of faculty as perceived by students through a comparision of 
response by sex to the ratio of sex in a faculty member's current year 
class enrolment. This sort of study would be infinitely more re- 
warding than a listing of preferences of Greek organizations, dormitory 
residency, etc. I believe that these recommendations are in line with 
the suggestions made by our computer utiliazation expects on how to use 
this sophisticated tool. 

2309 - This individual said that student comments are quite valuable 
". . .but obtained directly from students rather than in this form." 

2310 - No comments. 

2304 - (1.) Your classification of students largely not useful, i.e., 
dorm, fraternity, etc. (2) Response should be limited to current term. 
Otherwise one doesn't know how to evaluate students response. (3) I 
received a response for a course I have never taught. 

2103 - (1) Students are our "consumers" so we should pay stock to this 
fact. (2) Faculty should not be totally submissive to demands but 
should be aware of opinion and continusally strive to perfect the teach- 
ing-learning process. 

2206 - I was criticized rather sever ly for courses I haven't even taught 
at Austin College. This type of evaluation is prying, just as if one 
prof were going around grilling students about another prof. It is 
unethical and violates the student-professor relationship. (This comment 
was signed, but the signature has been deleted by the experimentor.) 



2226 - No comments. 
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APPENDIX E (cont.) 



Code Number and Comments 

2138 - I would enjoy receiving the narrative portion of the comments. 

2209 - (Concerning question number 20) I seriously doubt the possibility 
or feasibility of a reliable correction factor. The professors will 
compare themselves and their courses in the others anyway. The question- 
naire and results cf evaluation at least seem to be strying to get us 
to do more what students want. Perhaps this helps some and inhibits others?! 
Too many variables-- 

2104 - No comments. 

2211 - Good work David - and courageous work! This is a real contri- 
bution to Austin College! Thank You! (I do wish we could include some 
sort of VBascoe Unit" factor. . ." 

2105 - Please send student comments. 

2212 - Use a ratio of; student exposure in number and 

Number of years here 

various student responses 
positive, negative 

2312 - No comments. 

2101 - No comments. 

2320 - No comments. 

2124 - The fact that you send me comments about my performance in a 
course I have never taught makes me question the accuracy of the rest 
of the information. 

2240 - (1.) What do the numbers 339 signify on the History of Jazz 
comments? 

(2.) I consciously discourage students from taking Music 11 - in fact 
it is a department policy. Those who do not get the message find they 
are in the wrong course by the very demands of skills in music which 
they do not possess. 

(3.) I find the information interesting but of little value to me or 
my course structures. 

) Was applied music ever considered as course work? If not, why not? 
Students receive course credit for it. 

(5.) Lack of negative comments may help the ego but is of no assistance 
in growth. 

(6.) Teaching effectiveness is poorly measured when it depends on general 
"recommendations" by the student body. 

(7.) The teachers who most affected my professional career would probably 
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APPENDIX E (cont.) 



Code Number and Comments 

2240 (cont.) - have not received my recommendation when I was under 
their instruction. It appears as though the type of teacher and the 
courses he is teaching interrelate too closely for the survey to be 
of any real help. 

2239 - Very valuable information. Excellent survey. 

2122 - The number of responses relating to specific courses appears 
small and the number of negative responses I received appears too 
small in the general response section. This instrument is a significant 
improvement over past sampling and I think will lead to positive 
changes in student-teacher relationships. However, some faculty will 
(and have) reacted very defensively - some perhaps with cause. The in- 
strument appears to measure extremes (with the exception of student 
comments) which I feel is a short-coming in evaluation. 

2121 - No comments. 

2234 - The responses should be limited to actual experience in a 
course. A student's bad (or good) opinion of a professor based on 
hcrssay or experience with him on a committee or as an administrator 
is completely worthless in evaluating his teaching ability. 

2233 - No comments. 

2303 - Would like to receive student comment sheet. 

2232 - No comments. 

2230 - The problem is not the value of student opinion, the problem 
is the instrument used. This evaluation gets responses only from 
those who react strongly to your teaching, positively or negatively. 
Moreover it doesn't distinguish between heresay or brief encounters 
and experiences of your courses. The comments are too brief to be 
helpful - you merely learn that the student likes or dislikes you. 

The superficiality is indicated by the fact that I received comments 
on a number of courses I don't teach, some even in departments where 
I do not teach. The wording of this evaluation illustrates that more 
care should be taken. E.g., question 22--I think the comments provide 
s Qme information that will help me to improve my teaching but that 
information is not nearly precise enough for corrective action. But 
question 22 is worded ". . .for any kind of corrective action. . ." 

I would logically, love to argue. But I do not want to say the form 
is adequate, so I marked disagree. When you must do this, the form 
is faulty. 
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APPENDIX E (cont.) 



Code Number and Comments 

2106 - . . .a far more sophisticated (weighted) calculation might be 
valuable. 

2107 - There were not enough students evaluating me since I only 
taught a Jan-Term with 9 students. If I had taught a regular course 

this info would be very helpful, (signature deleted by the experimentor . ) 

2213 - Since my own scale of professional ,, success ,t rests almost 
wholly with colleague evaluation (when A. C. does not tabulate), my 
very flattering student profile leaves me as ambivolent as ever-- 
is it really my ability or only my personality which assists them in 
comprehending and enjoying and working hard in my courses? I could 
never be sure of this at my previous job either where I was "gembook 
professor of the year" for two out of five years! So what? My book, 
on the other hand is a scolarly and sales flop!! 

2316 - I would like to see the individual student comments. 

2215 - In format, this packet of information is much "better" (usable, 
meaningful, understandable) than was last years computer print-out. 

I have no objection to the projects continuation with proper safe- 
guards for confidentiality always applied. 

2317 - No comments. 

2216 - No comments. 

2217 - No comments. 

2318 - The positive and negative course evaluations are too ambiguous 
to be of much help. • .it would be useful to know how students reacted 
positively or negatively who--never had the course in question--just 
"finished" the course of "finished" it some semesters ago AA are reacting 
partly on the basis of factors (relitively) unrelated to the course 
(personal dislike, experience in other courses with same prof., etc.) 
are majors in various fields. The breakdown by sex, dorm, frat, etc., 
is really somewhat useless to me--though I recognize that it might be 
relevant to your own purposes in formulating the study. 

2218 - Appreciated the effort maintained in deepening the interviews 
open and student initiated . If there are other factors which the deans 
and other administrators use in evaluating faculty members, they should 
also be included on the student questionnaire to get multilpe input. 

Wish the students were broken down by majors, grade-point averages, 
urban or rural backgrounds, and goal orientations as well as by male, 
female, fraternity, etc. 



2219 - No comments. 
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APPENDIX E (cont.) 



Code Number and Comments 

2111 - I would like very much to have the graduate students included 
in this study because this is where we have our real contact with students, 
(signature deleted by experiraentor o ) 

2243 - No comments. 

2313 - I have to question the validity of the whole process since my 
name is associated with at least one course I have never taught. 

2306 - No comments. 

2245 - My main objection to this questionnaire is that it really does 
not help one change or correct his teaching performance. One knows 
that a certain number of students "recommend" him and others "don't 
recommend" but he has no idea what is their critera for "recommending" 
a teacher nor does he know the make-up of the groups themselves. For 
example : : he knew percentagely that majors recommended him and non- 
majors didn't then he might conclude he needs to make the course more 
appealing to non-majors. As it is he doesn't know which way to go. 

Another case would be if his advises recommended him but his students 
didn't this might indicate something to him and give him information 
for evaluating his teaching. As the information stands I find it so 
general as to be useless in helping me correct my teaching. Another 
problem is that if one taught a large class/freshman and had either 
very good or a very bad course this would affect the print-out for the 
next few years making judging of improvement or non-improvement almost 
impossible. I think I would welcome and find useful a student evaluation 
that provided foundations that I could make use of in evaluating and 
improving my teaching. 

2244 - No comment. 

2305 - There is no way one can judge whether one has improved his 
course because no distinction is made between the same course taught 
in different semesters. For example it's possible the 25 negative 
choices in a course could be from course X in Fall 1971 and anot from 
the same course taught in Fall 1972. 

2242 - Eventhough you try to achieve anonimiety on this, I have been 
able to identify several of the student comments because of the phrasing. 
Since I have fairly detail feedback in most courses and several types of 
student evaluations of my courses I did not find this to be particularly 
helpful. It confirmed the trends already will be established both 
positive and negative. I question the particular value of this "out of 
the situation" type evaluations. It has ics value but like every piece 
of "neutral" data it can be used for many purposes both constructive 
and destructive. 

2125 - No comments. 






O 

ERIC 



74 



* 



70 



APPENDIX E (cont.) 



Code Number and Comments 

2241 - I appreciate the purpose of such an effort as this and cannot 
fully appreciate the energy and concern of which such an effort 
requires of David Heyn and his assistants. I believe some correction 
or comparative factors would be beneficial, a wider range of courses 
and students within the same course would be better. 

2220 - Res Items 11 and 23. Response to 11 based on response to 23 
in my case. Items 20 and 21. Responses based on non-publicity of 
information except in same form as this. I'm not necessarily inter- 
ested in comparing myself with others (but perhaps knowing the three 
to five most positive faculty might be useful). But I would be inter- 
ested in positive "45" as based on a quotient of 150 or 750 student 
contacts. Along this line, I suspect it is of little value to get 
student recs on the basis of hearsay (see negative comments), rather 
suggest that it is not necessary perhaps to fill in all the blanks 
under the positive and negative recs. The data figure that pleased 
me most (of course, on the positive side) was that I was second among 
the entire faculty in positive responses from seniors. 

2322 — I didn't teach Chemo 31 or 32 which you had listed as course 
choices. Physical Science 13 and 14 was the same course given fall and 
spring. I thought the course went very well in the fall and this seemed 
to be backed by a student evaluation which I passed out. I felt that 
the course did not go well in the spring. I am unclear why it went 
well one semester and not the second since it was the same course. 
Unfortunately your data doesn't help this sort out. 

223 - No comments. 

2222 - No comments. 



2118 - No comments. 

2117 - Were there any positive statements cr just negative ones? 

2229 - Student opinion is valuable, but this particular method of 
getting it is, in my opinion almost completely worthless. The questions 
are insane, loaded, restricted and/or inadequate--a waste of a lot 
of time Bunk! Bunk! Bunk! (signature deleted by experimentor. ) 

2227 - In qualification of questions marked: Explanations of methods 

and original questionnaire (like Heyn's 6/16/72) are essential to 
interpreting this data. Last year, without it, the data was unclear (9). 

For what this method can show, it appears reliable. As extreme reactions 
it tends to show the kind of information that I already have. (2, 3, 8) 
Although this is valuable, if well done, as a part of evaluative 
judgements by myself or others. It would be very inadequate as the 
only are major instrument. A survey of students in a single professors 
courses would give another range of information for instance (10, 14, 20, 21). 
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APPENDIX E (cont.) 



Code Number and Comments 

2116 - Although helpful, I do not feel it is an adequate sample of the 
students with whom I have contacts 

2302 - No comments* 

2115 - The confidentiality of this research is very suspect* Four 
courses were listed on my ratings which I have never taught* This 
prompts me to wonder how accurate the results are and how well the 
interviews were conducted* Interesting project* It does provide 
some helpful evaluations from student perspectives* Good luck* 

2113 - No comments* 

2323 - No comments* 



